Skip to content

Add IO runtime handle to postgres connector#549

Merged
phillipleblanc merged 1 commit intodatafusion-contrib:mainfrom
alexanderbianchi:bianchi/runtimeIO
Feb 26, 2026
Merged

Add IO runtime handle to postgres connector#549
phillipleblanc merged 1 commit intodatafusion-contrib:mainfrom
alexanderbianchi:bianchi/runtimeIO

Conversation

@alexanderbianchi
Copy link
Contributor

@alexanderbianchi alexanderbianchi commented Feb 19, 2026

Summary

Adds an optional with_io_runtime(handle) builder method to PostgresConnectionPool, allowing callers to route connection acquisition (and the background socket-IO task that bb8 spawns internally) onto a dedicated tokio IO runtime.

This is the IO runtime segregation follow-up noted in #548.

Problem

When a query engine runs DataFusion plans and Postgres IO on the same tokio runtime, CPU-bound work (plan optimization, aggregation, shuffles) starves IO tasks, causing connection timeouts, health-check failures, and unpredictable tail latency. This is the same class of issue fixed in #450 (non-blocking connection check), but at the connection-pool level rather than the DNS-lookup level.

Solution

  • Add io_handle: Option<Handle> to PostgresConnectionPool
  • Add with_io_runtime(handle: Handle) -> Self builder method
  • When present, pool.get_owned() is dispatched via handle.spawn(...) so the connection and its background task land on the IO runtime
  • When absent, behavior is identical to today (run_async_with_tokio / direct .await)

The returned Client communicates with its background task via channels, so it works correctly from any runtime after acquisition.

Why this pattern

DataFusion's own documentation explicitly recommends runtime segregation:

"If your system does not fully utilize either the CPU or network bandwidth during execution, or you see significantly higher tail (e.g. p99) latencies responding to network requests, it is likely you need to use a different Runtime for DataFusion plans."

datafusion/core/src/lib.rs § Thread Scheduling

The pattern of accepting an optional Handle at construction time is converging across the ecosystem:

Crate API PR / Doc
object_store SpawnedReqwestConnector::new(handle) arrow-rs-object-store#332 (merged into 0.12.1)
DataFusion Separate-threadpool example datafusion#16331 (merged)
DataFusion Issue: move CPU off tokio datafusion#13692
DataFusion Issue: document runtime separation datafusion#12393

Changes

  • core/src/sql/db_connection_pool/postgrespool.rs
    • New error variant IoRuntimeError (wraps tokio::task::JoinError)
    • New field io_handle: Option<Handle> on PostgresConnectionPool
    • New builder method with_io_runtime(handle)
    • connect() and connect_direct() dispatch through the handle when present
  • core/tests/postgres/mod.rs
    • Integration test that creates a separate multi-thread IO runtime, constructs a pool with with_io_runtime(), and runs a query end-to-end

Usage

// Create a dedicated IO runtime
let io_runtime = tokio::runtime::Builder::new_multi_thread()
    .worker_threads(4)
    .enable_all()
    .build()?;

// Pass the handle to the pool — all connection background tasks
// will now run on io_runtime instead of the caller's runtime
let pool = PostgresConnectionPool::new(params)
    .await?
    .with_io_runtime(io_runtime.handle().clone());

Without with_io_runtime(), behavior is unchanged.

Test plan

  • cargo build --features postgres compiles
  • New test_postgres_io_runtime_segregation integration test verifies the IO-handle path end-to-end
  • Existing tests unaffected (no behavioral change when io_handle is None)

@alexanderbianchi
Copy link
Contributor Author

cc: @phillipleblanc who reviewed the other postgresSQL PR

Copy link
Collaborator

@phillipleblanc phillipleblanc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me, thanks!

@phillipleblanc phillipleblanc merged commit 3db3f59 into datafusion-contrib:main Feb 26, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants